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Claims 

1 . An imaging apparatus comprising: 

means for integration of spatial, spectral and temporal features, and 
optionally other physiologic data or calibration data, in a spectral and temporal 
5 multimodal imaging system for the evaluation of biological systems and stimuli; 

and 

means for fusing a thermal image or other imaging modalities and 
hyperspectral data cube for assessment of biological processes. 

2. The imaging apparatus of claim 1 wherein the integrated features are two or 
10 more of visible or infrared hyperspectral images, visible or infrared brightfield 

images, thermal images, fluorescence images, Raman images, or combinations 
thereof. 

3. The imaging apparatus of claim 1 further comprising: 
a specific UV, visible or infrared light source, and 

1 5 means for collecting two or more of visible or infrared hyperspectral images, 

visible or infrared brightfield images, thermal images, fluorescence images, 
Raman images, or combinations thereof 

4. The imaging apparatus of claim 1 wherein the light source can be controlled 
^ to limit the illumination to only a single or group of spectral bands. 

20 5. The imaging apparatus of claim 1 wherein the multimodal imaging systems 
includes a collection optic that is an endoscope. 
6. The imaging apparatus of claim 1 further comprising inputs for signals from 
instruments monitoring a biological function to allow the synchronization of the 
collection of images to the biological process. 
25 7. The imaging apparatus of claim 1 further comprising: 

on-board CCD chip filter and processing electronics to perforai data 
operations prior to transmission of signal to increase processing speed; and 
inputs for a physiological monitoring system. 
8. The imaging apparatus of claim 1 further comprising: 
30 sensors for image fusion and alignment; 

alignment mechanisms for aligning multiple optical paths; 
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normal color video for displaying data and for camera alignment and aiming; 

and 

a storage system for recording and logging of real-time or near real-time 
output. 

9. The imaging apparatus of claim 1 further comprising means for stabilizing 
and integrating imagery and data sets in a temporally and geometrically dynamic 
environment in such a way that the integrated data sets can be interpreted. 

10. The imaging apparatus of claim 1 further comprising an image registration 
device to maintain image centering. 

11. The imaging apparatus of claim 1 further comprising a reference frame 
device that is visible in the images across multiple spectra as well as in thermal 
images for multi-modal fusion. 

1 2. The imaging apparatus of claim 1 further comprising: 

a tracked imaging apparatus comprising a means of creating a unified spatial 
mosaic of individual hyperspectral image cubes taken at various locations and 
times. 

13. A method comprising the steps of: 

integrating spatial, spectral and temporal features, and optionally physiologic 
data, with a spectral and temporal multimodal imaging system for the evaluation 
of biological systems and stimuli; and 

fusing a thermal image or other imaging modalities with hyperspectral 
images. 

14. The method of claim 13 further comprising the step of interlacing thermal 
images and hyperspectral images with a time-dependent reference. 

15. The method of claim 13 further comprising the step of selecting a 
wavelength to maximize diagnostic information for a specific tissue state or 
anticipated end diagnostic goal. 

16. The method of claim 13 wherein the selection step is perforaied by 
multivariate image and spectral processing algorithms to extract information 
from the plurality of images and spectra for real-time or near real-time 
assessment. 

17. The method of claim 13 wherein the target is skin. 
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1 8. The method of claim 1 3 wherein the target is an organ or a tissue 

19. The method of claim 13 wherein the analysis step determines the spatial 
distribution of oxyhemoglobin, deoxyhemoglobin and methemoglobin. 

20. The method of claim 19 wherein the analysis step detemiines a total 
5 hematocrit by calculating the area under the oxyhemoglobin, deoxyhemoglobin 

and methemoglobin spectrum or the intensity at their respective wavelengths, 
and a total blood volume by pairing a total hematocrit with perfusion data from a 
broadband thermal camera. 

21. The method of claim 19 wherein the analysis step determines a state of 
10 perfusion, demonstrates state of oxygen extraction, neurohumoral, physiological 

or pathological circumstances and in specific spatial patterns. 

22. The method of claim 13 further comprising the step of gating each image of 
a hyperspectral image set as well as the adjunct thermal images to a biological 
function such as the cardiac cycle, breathing, pulse, or muscle contraction, to 

1 5 allow data collection from moving samples. 

23. The method of claim 13 wherein the analysis step identifies and classifies 
chemical species other than hemoglobins. 

24. The method of claim 13 further comprising the steps of: 

^ registering multiple images in the hyperspectral cube acquired at various 

20 wavelengths and at different times so that each con-esponding pixel in every 

plane images the same point in space. 

incorporating a high spatial and temporal resolution imaging device that is 
used as a reference in which each spectral image in the cube is registered. 

25. The method of claim 24, further comprising the step of utilizing automatic 
25 feature extraction techniques to localize landmarks throughout the hyperspectral 

imaging cube. 

26. The method of claim 24, further comprising the step of registering and 
analyzing multiple hyperspectral image sets taken over time. 

27. The method of claim 24, further comprising the step of registering a 
30 hyperspectral image set with a three dimensional spatial medical image. 

28. The method of claim 27 wherein the fusion is performed with a real-time or 
near real-time hyperspectral imaging device and a 3D medical image. 
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29. The method of claim 24, further comprising the step of performing real-time 
or near real-time visualization of the full spectral information at a given location 
by registering a real-time or near real-time trackable instrument with the images. 

30. The method of claim 27 wherein the three dimensional spatial medical image 
is selected from the group consisting of MR, CT, PET, SPECT, ultrasound, or 
combinations thereof 
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